Skip to content

Make Iceberg to Spark compatibility tests more resilient to retries #21310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 3, 2024
Merged

Make Iceberg to Spark compatibility tests more resilient to retries #21310

merged 1 commit into from
Apr 3, 2024

Conversation

@nineinchnick
Copy link
Member

@nineinchnick nineinchnick commented Mar 28, 2024

Description

Fixes #21304

Additional context and related issues

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

@cla-bot cla-bot bot added the cla-signed label Mar 28, 2024
@findinpath findinpath requested a review from pajaks March 29, 2024 07:12
@nineinchnick nineinchnick deleted the iceberg-compatibility-pt-resiliency branch March 29, 2024 10:35
@nineinchnick nineinchnick restored the iceberg-compatibility-pt-resiliency branch April 2, 2024 09:14
@nineinchnick nineinchnick reopened this Apr 2, 2024
@findinpath
Copy link
Contributor

findinpath commented Apr 2, 2024

The failure below seems related to your changes: https://github.com/trinodb/trino/actions/runs/8520349910/job/23336628885?pr=21310

2024-04-02T09:29:48.1315619Z tests               | 2024-04-02 15:14:48 SEVERE: Failure cause:
2024-04-02T09:29:48.1318417Z tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20240402_092948_00559_2drgq): line 1:18: Table 'hive.default."iceberg.default.test_register_table_with_comments_parquet_vwe36pl231$metadata_log_entries"' does not exist
2024-04-02T09:29:48.1321163Z tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
2024-04-02T09:29:48.1322674Z tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
2024-04-02T09:29:48.1324276Z tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
2024-04-02T09:29:48.1325739Z tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
2024-04-02T09:29:48.1326884Z tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
2024-04-02T09:29:48.1328219Z tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
2024-04-02T09:29:48.1330073Z tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
2024-04-02T09:29:48.1331374Z tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
2024-04-02T09:29:48.1332978Z tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
2024-04-02T09:29:48.1334485Z tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
2024-04-02T09:29:48.1336308Z tests               | 	at io.trino.tests.product.iceberg.util.IcebergTestUtils.getLatestMetadataLocation(IcebergTestUtils.java:31)
2024-04-02T09:29:48.1338689Z tests               | 	at io.trino.tests.product.iceberg.TestIcebergSparkCompatibility.testRegisterTableWithComments(TestIcebergSparkCompatibility.java:2561)
2024-04-02T09:29:48.1402952Z tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2024-04-02T09:29:48.1404500Z tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
2024-04-02T09:29:48.1408753Z tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
2024-04-02T09:29:48.1410267Z tests               | 	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
2024-04-02T09:29:48.1411458Z tests               | 	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
2024-04-02T09:29:48.1412705Z tests               | 	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
2024-04-02T09:29:48.1414115Z tests               | 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
2024-04-02T09:29:48.1415543Z tests               | 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
2024-04-02T09:29:48.1417037Z tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2024-04-02T09:29:48.1418686Z tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2024-04-02T09:29:48.1420614Z tests               | 	at java.base/java.lang.Thread.run(Thread.java:1570)
2024-04-02T09:29:48.1423162Z tests               | Caused by: java.sql.SQLException: Query failed (#20240402_092948_00559_2drgq): line 1:18: Table 'hive.default."iceberg.default.test_register_table_with_comments_parquet_vwe36pl231$metadata_log_entries"' does not exist
2024-04-02T09:29:48.1425765Z tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
2024-04-02T09:29:48.1427225Z tests               | 	at io.trino.jdbc.TrinoResultSet.getColumns(TrinoResultSet.java:318)
2024-04-02T09:29:48.1429139Z tests               | 	at io.trino.jdbc.TrinoResultSet.create(TrinoResultSet.java:61)
2024-04-02T09:29:48.1430442Z tests               | 	at io.trino.jdbc.TrinoStatement.internalExecute(TrinoStatement.java:262)
2024-04-02T09:29:48.1431699Z tests               | 	at io.trino.jdbc.TrinoStatement.execute(TrinoStatement.java:240)
2024-04-02T09:29:48.1433471Z tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:128)
2024-04-02T09:29:48.1435100Z tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
2024-04-02T09:29:48.1436096Z tests               | 	... 22 more
2024-04-02T09:29:48.1437885Z tests               | 	Suppressed: java.lang.Exception: Query: SELECT file FROM "iceberg.default.test_register_table_with_comments_parquet_vwe36pl231$metadata_log_entries" ORDER BY latest_sequence_number DESC LIMIT 1
2024-04-02T09:29:48.1440252Z tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
2024-04-02T09:29:48.1441401Z tests               | 		... 23 more

@nineinchnick nineinchnick force-pushed the iceberg-compatibility-pt-resiliency branch from 9de9711 to 57a2094 Compare April 2, 2024 10:38
@nineinchnick
Copy link
Member Author

nineinchnick commented Apr 2, 2024

@findinpath fixed

@nineinchnick nineinchnick force-pushed the iceberg-compatibility-pt-resiliency branch from 1060cd7 to 02654a7 Compare April 2, 2024 14:03
findinpath
findinpath reviewed Apr 2, 2024
View reviewed changes
Copy link
Contributor

@findinpath findinpath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there. A few nits still needed.

@nineinchnick nineinchnick force-pushed the iceberg-compatibility-pt-resiliency branch from 02654a7 to 4182222 Compare April 3, 2024 07:19
@nineinchnick nineinchnick force-pushed the iceberg-compatibility-pt-resiliency branch from 4182222 to 0771e0c Compare April 3, 2024 09:23
@ebyhr ebyhr merged commit 9d28476 into trinodb:master Apr 3, 2024
@github-actions github-actions bot added this to the 444 milestone Apr 3, 2024
@mosabua mosabua mentioned this pull request Apr 3, 2024
Merged
@nineinchnick nineinchnick deleted the iceberg-compatibility-pt-resiliency branch April 4, 2024 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ebyhr ebyhr ebyhr approved these changes

@pajaks pajaks Awaiting requested review from pajaks

@hashhar hashhar Awaiting requested review from hashhar

+1 more reviewer

@findinpath findinpath findinpath approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@nineinchnick nineinchnick

Labels

cla-signed

Milestone

444

Development

Successfully merging this pull request may close these issues.

TestIcebergSparkCompatibility.testRegisterTableWithShowCreateTable flaky

4 participants

@nineinchnick @findinpath @ebyhr @hashhar